Opacity filter for display device

ABSTRACT

An optical see-through head-mounted display device includes a see-through lens which combines an augmented reality image with light from a real-world scene, while an opacity filter is used to selectively block portions of the real-world scene so that the augmented reality image appears more distinctly. The opacity filter can be a see-through LCD panel, for instance, where each pixel of the LCD panel can be selectively controlled to be transmissive or opaque, based on a size, shape and position of the augmented reality image. Eye tracking can be used to adjust the position of the augmented reality image and the opaque pixels. Peripheral regions of the opacity filter, which are not behind the augmented reality image, can be activated to provide a peripheral cue or a representation of the augmented reality image. In another aspect, opaque pixels are provided at a time when an augmented reality image is not present.

CLAIM OF PRIORITY

This application is a continuation of co-pending U.S. patent applicationSer. No. 15/043,291, entitled “OPACITY FILTER FOR DISPLAY DEVICE,” filedFeb. 12, 2016, published as US 2016/0171779 on Jun. 16, 2016, and issuedas U.S. Pat. No. 9,911,236on Mar. 6, 2018, which is a continuation ofU.S. application Ser. No. 14/605,701, entitled “OPACITY FILTER FORDISPLAY DEVICE,” filed Jan. 26, 2015, and published as US 2015/0193984on Jul. 9, 2015 and issued as U.S. Pat No. 9,286,730 on Mar. 15, 2016,which in turn is a divisional application of U.S. patent applicationSer. No. 12/887,426, entitled “OPACITY FILTER FOR DISPLAY DEVICE,” filedSep. 21, 2010, published as US 2012/0068913 on March 22, 2012 and issuedas U.S. Pat. No. 8,941,559 on Jan. 27, 2015, all of which areincorporated herein by reference in their entirety.

BACKGROUND

Head-mounted displays can be used in various application, includingmilitary, aviation, medicine, video gaming, entertainment, sports, andso forth. See-through head-mounted displays allow the user to observethe physical world around him or her, while optical elements add lightfrom one or two small micro-displays into the user's visual path, toprovide an augmented reality image. The augmented reality image mayrelate to a real-world scene which represents an environment in which auser is located. However, various challenges exist in providing anaugmented reality image which is realistic and which can represent afull range of colors and intensities.

SUMMARY

An optical see-through head-mounted display device is provided. Thehead-mounted display device uses an opacity filter to selectively removelight from a real-world scene which reaches a user's eye. For example,the filter may block light based on a shape of an augmented realityimage to avoid the augmented reality image being transparent. Further,an eye tracking component may be used to adjust a position of theaugmented reality image and increased-opacity pixels of the opacityfilter.

In one embodiment, an optical see-through head-mounted display (HMD)device includes a see-through lens extending between a user's eye and areal-world scene when the display device is worn by the user. Thesee-through lens has an opacity filter with a grid of pixels which canbe controlled to adjust their opacity, from a minimum opacity levelwhich allows a substantial amount of light to pass, to a maximum opacitylevel which allows little or no light to pass. The see-through lens alsohas a display component. The device further includes at least oneaugmented reality emitter, such as a micro-display, which emits light tothe user's eye using the display component, where the light representsan augmented reality image having a shape. The device further includesat least one control which controls the opacity filter to provide anincreased opacity for pixels which are behind the augmented realityimage, from a perspective of the user's eye. The increased-opacitypixels are provided according to the shape of the augmented realityimage.

An eye tracking component can be provided to track a location of theuser's eye relative to a frame, so that the position of theincreased-opacity pixels and/or the augmented reality image can beadjusted when there is movement of a frame on which the HMD device iscarried. In this way, the identified pixels and the augmented realityimage can be shifted based on movement of the frame, while theirregistration to one another is maintained.

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the description.This summary is not intended to identify key features or essentialfeatures of the claimed subject matter, nor is it intended to be used tolimit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like-numbered elements correspond to one another.

FIG. 1 depicts an example embodiment of an optical see-through HMDdevice with an augmented reality capability.

FIG. 2 depicts a system diagram of the HMD device of FIG. 1.

FIG. 3A depicts a process for providing an augmented reality image inthe HMD device of FIG. 1.

FIG. 3B depicts details of step 306 of FIG. 3A.

FIG. 4A depicts an example configuration of an opacity filter based on ashape of the augmented reality image of 104 of FIG. 1 or FIG. 4C.

FIG. 4B depicts the example real-world scene 120 of FIG. 1.

FIG. 4C depicts the example augmented reality image 104 of FIG. 1.

FIG. 4D depicts the example image 132 of FIG. 1 which is seen by a user.

FIG. 5 depicts an opacity filter with increased-opacity regions, toprovide the configuration of the opacity filter of FIG. 4A.

FIG. 6 depicts a variation of the example image of FIG. 1 which wouldresult without the opacity filter.

FIG. 7A depicts an example implementation of the display device of FIG.1, as worn on a user's head.

FIG. 7B depicts further details of the HMD device of FIG. 7A.

FIG. 7C depicts an alternative implementation of the display device ofFIG. 1, as worn on a user's head, where the eye tracking component isdirectly on the front eye glass frame.

FIG. 8A1 depicts a registration of a real-world image and anincreased-opacity region of an opacity filter when the user's eye is ina first location relative to a frame of the HMD device.

FIG. 8A2 depicts a front-facing view of the real-world scene element 800of FIG. 8A1.

FIG. 8A3 depicts a front-facing view of the opacity filter region 804 ofFIG. 8A1.

FIG. 8A4 depicts a front-facing view of the augmented reality imageregion 805 of FIG. 8A1.

FIG. 8B1 depicts a registration of a real-world image and anincreased-opacity region of an opacity filter when the user's eye is ina second location relative to a frame of the HMD device.

FIG. 8B2 depicts a front-facing view of the real-world scene element 800of FIG. 8B1.

FIG. 8B3 depicts a front-facing view of the opacity filter region 806 ofFIG. 8B1.

FIG. 8B4 depicts a front-facing view of the augmented reality imageregion 807 of FIG. 8B1.

FIG. 9A1 depicts a registration of an augmented reality image and anincreased-opacity region of an opacity filter, at a center of anaugmented reality display region of a field of view of a user's eye.

FIG. 9A2 depicts a front-facing view of the opacity filter region 902 ofFIG. 9A1.

FIG. 9A3 depicts a front-facing view of the augmented reality imageregion 900 of FIG. 9A1.

FIG. 9B1 depicts a registration of an augmented reality image and anincreased-opacity region of an opacity filter, at a peripheral boundaryof the augmented reality display region of FIG. 9A1.

FIG. 9B2 depicts a front-facing view of the opacity filter region 920 ofFIG. 9B1.

FIG. 9B3 depicts a front-facing view of the augmented reality imageregion 922 of FIG. 9B1.

FIG. 9C1 depicts a gradual change in opacity as a function of a distancefrom a peripheral boundary of a field of view of a user's eye.

FIG. 9C2 depicts an opacity filter region with a non-faded portion 931and successively faded portions 932, 933 and 934, with fading between 0and d1 in FIG. 9C1.

FIG. 9C3 depicts an opacity filter region with a non-faded portion 941and successively faded portions 942, 943 and 944, with fading between 0and d3 in FIG. 9C1.

FIG. 9C4 depicts an opacity filter region with a non-faded portion 951and successively faded portions 952, 953 and 954, with fading between d4and d5 in FIG. 9C1.

FIG. 9D1 depicts a registration of an augmented reality image and anincreased-opacity region of an opacity filter, at a peripheral boundaryof the augmented reality display region of FIG. 9A1, where an additionalregion of increased opacity is provided in a second, peripheral regionof the field of view.

FIG. 9D2 depicts a front-facing view of the opacity filter regions 920and 924 of FIG. 9D1.

FIG. 9D3 depicts a front-facing view of the augmented reality imageregion 900 of FIG. 9D1.

FIG. 9E1 depicts a registration of a first portion of an augmentedreality image and an increased-opacity region of an opacity filter, at aperipheral boundary of the augmented reality display region of FIG. 9A1,where an additional region of increased opacity is provided in a second,peripheral region of the field of view to represent a second, cutoffportion of the augmented reality image.

FIG. 9E2 depicts a front-facing view of the opacity filter regions 926and 928 of FIG. 9E1.

FIG. 9E3 depicts a front-facing view of the augmented reality imageregions 922 and 923 of FIG. 9E1.

FIG. 9F1 depicts an increased-opacity region of an opacity filter in asecond, peripheral region of a field of view, at a time when noaugmented reality image is provided.

FIG. 9F2 depicts a front-facing view of the opacity filter region 960 ofFIG. 9F1.

FIG. 9F3 depicts a front-facing view of the augmented reality image ofFIG. 9F1.

DETAILED DESCRIPTION

See-through head-mounted displays (HMDs) most often use optical elementssuch as mirrors, prisms, and holographic lenses to add light from one ortwo small micro-displays into the user's visual path. By their verynature, these elements can only add light, but cannot remove light. Thismeans a virtual display cannot display darker colors—they tend towardstransparent in the case of pure black—and virtual objects such asaugmented reality images, seem translucent or ghosted. For compellingaugmented reality or other mixed reality scenarios, it is desirable tohave the ability to selectively remove natural light from the view sothat virtual color imagery can both represent the full range of colorsand intensities, while making that imagery seem more solid or real. Toachieve this goal, a lens of a HMD device can be provided with anopacity filter which can be controlled to selectively transmit or blocklight on a per-pixel basis. Control algorithms can be used to drive theintensity and/or color of the opacity filter based on the augmentedreality image. The opacity filter can be placed physically behind anoptical display component which introduces the augmented reality imageto the user's eye. Additional advantages can be obtained by having theopacity filter extend beyond a field of view of the augmented realityimage to provide peripheral cues to the user. Moreover, peripheral cues,or a representation of the augmented reality image, can be provided bythe opacity filter even in the absence of an augmented reality image.

FIG. 1 depicts an example embodiment of an optical see-through HMDdevice with an augmented reality capability. The display device caninclude a see-through lens 108 which is placed in front of a user's eye,similar to an eyeglass lens. Typically, a pair of see-through lenses areprovided, one for each eye. The lens includes an opacity filter 106 andan optical display component 112 such as a beam splitter, e.g., ahalf-silvered mirror or other light-transmissive mirror. Light from areal world scene 120, such as a light ray 114, reaches the lens and isselectively passed or blocked by the opacity filter 106. The light fromthe real world scene which passes through the opacity filter also passesthrough the display component.

The opacity filter is under the control of an opacity filter controlcircuit 100. Meanwhile, an augmented reality emitter 102 emits a 2-Darray of light representing an augmented reality image 104 andexemplified by a light ray 110. Additional optics are typically used torefocus the augmented reality image so that it appears to originate fromseveral feet away from the eye rather than from about one inch away,where the display component actually is.

The augmented reality image is reflected by the display component 112toward a user's eye 118, as exemplified by a light ray 116, so that theuser sees an image 132. In the image 132, a portion of the real-worldscene 120, such as a grove of trees, is visible, along with the entireaugmented reality image 104, such as a flying dolphin. The usertherefore sees a fanciful image in which a dolphin flies past trees, inthis entertainment-oriented example. In an advertising oriented example,the augmented reality image can appear as a can of soda on a user'sdesk. Many other applications are possible. Generally, the user can wearthe HMD device anywhere, including indoors or outdoors. Various piecesof information can be obtained to determine what type of augmentedreality image is appropriate and where it should be provided on thedisplay component. For example, the location of the user, the directionin which the user is looking, and the location of floors, walls andperhaps furniture, when the user is indoors, can be used to decide whereto place the augmented reality image in an appropriate location in thereal world scene.

The direction in which the user is looking can be determined by trackinga position of the user's head using a combination of motion trackingtechniques and an inertial measure unit which is attached to the user'shead, such as via the augmented reality glasses. Motion trackingtechniques use a depth sensing camera to obtain a 3D model of the user.A depth sensing camera can similarly be used to obtain the location offloors, walls and other aspects of the user's environment. See, e.g., US2010/0197399, published Aug. 5, 2010, titled “Visual Target Tracking,”US 2010/0194872, published Aug. 5, 2010, titled “Body Scan,” and U.S.Pat. No. 7,515,173, issued Apr. 7, 2009, titled “Head Pose TrackingSystem,” each of which is incorporated herein by reference.

A portion of the real-world scene which is behind the augmented realityimage, from a perspective of the user's eye, is blocked by the opacityfilter from reaching the user's eye, so that the augmented reality imageappears clearly to the user. The augmented reality image may beconsidered to provide a primary display, while the opacity filterprovides a secondary display. The intensity and/or color of thesecondary display can be driven to closely match the imagery on theprimary display, enhancing the ability of the primary display toresemble natural light.

A tracking camera 122 can be used to identify a location of the user'seye with respect to a frame on which the HMD device is mounted. Theframe can be similar to conventional eyeglass frames, in one approach.See, e.g., FIGS. 7A and 7B for an example of a frame. Typically, such aframe can move slightly on the user's head when worn, e.g., due tomotions of the user, slipping of the bridge of the frame on the user'snose, and so forth. See FIGS. 8A1-8B4 for further details. By providingreal-time information regarding the location of the eye with respect tothe frame, the controller can control the opacity filter, and theaugmented reality emitter can adjust its image, accordingly. Forexample, the augmented reality image can be made to appear more stable,while a registration or alignment of increased-opacity pixels of theopacity filter and the augmented reality image is maintained. In anexample approach, the tracking camera 122 includes an infrared (IR)emitter 124 which emits IR light 128 toward the eye 118, and an IRsensor 126 which senses reflected IR light 130. The position of thepupil can be identified by known imaging techniques such as detectingthe reflection of the cornea. For example, see U.S. Pat. No. 7,401,920,titled “Head mounted eye tracking and display system” issued Jul. 22,2008 to Ophir et al., incorporated herein by reference. Such techniquescan locate a position of the center of the eye relative to the trackingcamera. Generally, eye tracking involves obtaining an image of the eyeand using computer vision techniques to determine the location of thepupil within the eye socket. Other eye tracking technique can use arraysof photo detectors and LEDs. With a known mounting location of thetracking camera on the frame, the location of the eye with respect toany other location which is fixed relative to the frame, such as theopacity filter 106 and the optical component 112, can be determined.Typically it is sufficient to track the location of one of the user'seyes since the eyes move in unison. However, it is also possible totrack each eye separately and use the location of each eye to determinethe location of the augmented reality image for the associatedsee-through lens.

In the example depicted, the tracking camera images the eye from a sideposition on the frame that is independent from the opacity filter andoptical component 112. However, other approaches are possible. Forexample, light used by the tracking camera could be carried via theoptical component 112 or otherwise integrated into the lens.

The opacity filter can be a see-through LCD panel, electrochromic film,or similar device which is capable of serving as an opacity filter. Sucha see-through LCD panel can be obtained by removing various layers ofsubstrate, backlight and diffusers from a conventional LCD. The LCDpanel can include one or more light-transmissive LCD chips which allowlight to pass through the liquid crystal. Such chips are used in LCDprojectors, for instance.

The opacity filter can be placed over or inside the lens. The lens mayalso include glass, plastic or other light-transmissive material. Theopacity filter can include a dense grid of pixels, where the lighttransmissivity of each pixel is individually controllable betweenminimum and maximum transmissivities. While a transmissivity range of0-100% is ideal, more limited ranges are also acceptable. As an example,a monochrome LCD panel with no more than two polarizing filters issufficient to provide an opacity range of about 50% to 80 or 90% perpixel, up to the resolution of the LCD. At the minimum of 50%, the lenswill have a slightly tinted appearance, which is tolerable. 100%transmissivity represents a perfectly clear lens. We can define an“alpha” scale from 0-100% where 0% is the highest transmissivity (leastopaque) and 100% is the lowest transmissivity (most opaque). The value“alpha” can be set for each pixel by the opacity filter control circuit.

A mask of alpha values can be used from a rendering pipeline, afterz-buffering with proxies for real-world objects. When we render a scenefor the augmented reality display, we want to take note of whichreal-world objects are in front of which augmented reality objects. Ifan augmented reality object is in front of a real-world object, then theopacity should be on for the coverage area of the augmented realityobject. If the augmented reality object is (virtually) behind areal-world object, then the opacity should be off, as well as any colorfor that pixel, so the user will only see the real-world object for thatcorresponding area (a pixel or more in size) of real light. Coveragewould be on a pixel-by-pixel basis, so we could handle the case of partof an augmented reality object being in front of a real-world object,part of an augmented reality object being behind a real-world object,and part of an augmented reality object being coincident with areal-world object.

Additional enhancements come in the form of new display types repurposedto use as opacity filters. Displays capable of going from 0% to 100%opacity at low cost, power, and weight are the most desirable for thisuse. Moreover, the opacity filter can be rendered in color, such as witha color LCD or with other displays such as organic LEDs, to provide awide field of view surrounding the optical component 112 which providesthe augmented reality image.

The opacity filter control circuit 100 can be a micro-processor, forinstance. The opacity filter control circuit 100 and the augmentedreality emitter 102 may communicate with the tracking camera 122. In oneoption, a central control (not shown) communicates with the trackingcamera 122, and is used to oversee the opacity filter control circuit100 and the augmented reality emitter 102. Appropriate wired or wirelesscommunication paths between the components 100, 102 and 122 can beprovided and integrated into the frame of the HMD device.

The resulting HMD device is relatively streamlined, compared to devicessuch as conventional LCD shutter glasses for active stereo 3D viewing,which typically require complex optics. These are glasses used inconjunction with a display screen to create the illusion of a 3D image.In the eyeglass lens, a liquid crystal layer can switch from beingtransparent to being opaque when a voltage is applied, so thateffectively one pixel per eye is provided. The glasses can be controlledby a wireless signal in synchronization with the refresh rate of thescreen. The screen alternately displays different perspectives for eacheye, which achieves the desired effect of each eye seeing only the imageintended for it. The HMD device provided herein has the ability tooperate as shutter glasses by controlling all pixels of the opacityfilter together to be transparent or opaque.

In another alternative, the HMD device can provide passive stereoscopicvision. Since the filters used in LCD panels are polarized, we canorient the LCD panels of the right and left lenses so that thepolarization is different by 90 degrees. This changes the behavior ofthe rotated LCD so that transmissivity and opacity are reversed. Avoltage applied results in transmissivity and no voltage applied resultsin opacity. For the non-rotated LCD, a voltage applied results inopacity and no voltage applied results in transmissivity.

An opacity filter such as an LCD has generally not been used in asee-through lens as described herein because at this near distance tothe eye, it is almost completely out of focus. However, this result isactually desirable for our purposes. A user sees the augmented realityimage with crisp color graphics via the normal HMD display usingadditive color, which is designed to be in focus. The LCD panel isplaced “behind” this display such that a fuzzy black border surroundsany virtual content, making it as opaque as desired. We convert the flawof natural blurring to expediently obtain the feature of anti-aliasingand bandwidth reduction. These are a natural result of using alower-resolution and out-of-focus image. There is an effective smoothingof the digitally-sampled image. Any digital image is subject toaliasing, where the discrete nature of the sampling causes errorsagainst the naturally analog and continuous signal, around thewavelengths of light. Smoothing means visually closer to the idealanalog signal. Although information lost to the low resolution is notrecovered, the resulting errors are less noticeable.

We optimize graphics rendering such that the color display and theopacity filter are rendered simultaneously and are calibrated to auser's precise position in space to compensate for angle-offset issues.Eye tracking can be employed to compute the correct image offset at theextremities of the viewing field. The opacity filter or mask canfurthermore be enlarged to cover the entire lens of the HMD device,extending beyond the display component of the augmented reality image ina central field of view. The opacity mask can also be rendered in color,either with a color LCD, or with other displays such as an organic LED(OLED), to provide a wide field of view surrounding the high-resolutionfocal area in the central field of view.

FIG. 2 depicts a system diagram of the HMD device of FIG. 1. The systemincludes the eye tracking camera 122, the augmented reality emitter 102and the opacity filter control circuit 100, which can communicate withone another via a bus 202 or other communication paths. The eye trackingcamera 122 includes a processor 212, a memory 214, an IR emitter 216, anIR sensor 218 and an interface 220. The memory 214 can containinstructions which are executed by the processor 212 to enable the eyetracking camera to perform its functions as described herein. Theinterface allows the eye tracking camera to communicate data to theaugmented reality emitter and the opacity filter control circuit, whichindicates the relative location of the user's eye with respect to theframe. The opacity filter control circuit can use the data to provide acorresponding offset to the pixels which have an increased opacity inthe opacity filter. Similarly, the augmented reality emitter can use thedata to provide a corresponding offset to the pixels which are used toemit the augmented reality image.

In another approach, it is sufficient for the eye tracking camera tocommunicate the eye location data to the augmented reality emitter, inwhich case the augmented reality emitter provides data to the opacityfilter control circuit to indicate which pixels of the opacity filtershould have an increased opacity. Or, the eye tracking camera cancommunicate the eye location data to the opacity filter control circuitwhich relays the data to the augmented reality emitter. In anotherpossibility, the opacity filter control circuit but not the augmentedreality emitter uses the eye location data, since changes in the pixelsof the opacity filter are more noticeable than changes in the augmentedreality image, due to the closeness of the opacity filter to the eye.

In any case, the augmented reality emitter can provide data to theopacity filter control circuit which indicates a shape of the augmentedreality image. The shape can be defined by a perimeter and the enclosedpoints. This data can be also used by the opacity filter control circuitto decide which pixels of the opacity filter should be provided with anincreased opacity, usually in correspondence with the size and shape ofthe augmented reality image.

The augmented reality emitter includes a processor 222, a memory 224, alight emitter 226 which emits visible light and an interface 228. Thememory 224 can contain instructions which are executed by the processor222 to enable the augmented reality emitter to perform its functions asdescribed herein. The light emitter can be a micro-display such as anLCD which emits a 2D color image in a small area such as one quarterinch square. The interface may be used to communicate with the eyetracking camera and/or the opacity filter control circuit.

The opacity filter control circuit 100 includes a processor 232, amemory 234, an opacity filter driver 236 and an interface 238. Thememory 234 can contain instructions which are executed by the processor232 to enable the opacity filter control circuit to perform itsfunctions as described herein. The opacity filter driver can drivepixels in the opacity filter 106 such as by addressing each pixel by arow and column address and a voltage which indicates a desired degree ofopacity, from a minimum level which is most light-transmissive level toa maximum level which is most opaque or least light-transmissive. Insome cases, a color of each pixel is set. The interface may be used tocommunicate with the eye tracking camera and/or the augmented realityemitter. The opacity filter control circuit communicates with theopacity filter 106 to drive its pixels.

One of more of the processors 212, 222 and 232 can be considered to becontrol circuits. Moreover, one or more of the memories 214, 224 and 234can be considered to be a tangible computer readable storage havingcomputer readable software embodied thereon for programming at least oneprocessor or control circuit to perform a method for use in an opticalsee-through HMD device as described herein.

The system may further components, discussed previously, such as fordetermining a direction in which the user is looking, the location offloors, walls and other aspects of the user's environment.

FIG. 3A depicts a process for providing an augmented reality image inthe HMD device of FIG. 1. At step 300, the eye tracking componentprovides data regarding the relative location of the eye. Generally,this can be performed several times per second. The data can indicate anoffset of the eye from a default location, such as when the eye islooking straight ahead. At step 302, the augmented reality emitterprovides data regarding size, shape and location (and optionally color)of an augmented reality image to the opacity filter control circuit. Thelocation data can be based on the data regarding the relative locationof the eye. The augmented reality image is an image which is set basedon the needs of an application in which it is used. For instance, theprevious example of a flying dolphin is provided for an entertainmentapplication. At step 304, the augmented reality emitter emits theaugmented reality image, so that it reaches the user's eye via one ormore optical components. Concurrently, at step 306, the opacity filtercontrol circuit drives pixels of the opacity filter, to provide anincreased opacity behind the augmented reality image. At decision step310, if there is a next augmented reality image, the process is repeatedstarting at step 300. If there is no next augmented reality image, theprocess ends at step 312.

The next augmented reality image can refer to the same augmented realityimage as previously provided, but in a different location, as seen bythe user, such as when the previous augmented reality image is moved toa slightly different location to depict movement of the augmentedreality image. The next augmented reality image can also refer to a newtype of image, such as switching from a dolphin to another type ofobject. The next augmented reality image can also refer to adding a newobject while a previously displayed object continues to be displayed. Inone approach, the augmented reality emitter emits video images at afixed frame rate. In another approach, static images are emitted andpersisted for a period of time which is greater than a typical videoframe period.

Step 314 optionally provides a gradual fade in the augmented realityimage, such as when it is near a boundary of an augmented realitydisplay region of a field of view. The augmented reality display regioncan be defined by the maximum angular extent (vertically andhorizontally) in the user's field of view in which the augmented realityimage is constrained, due to limitations of the augmented realityemitter and/or optical components 112. Thus, the augmented reality imagecan appear in any portion of the augmented reality display region, butnot outside the augmented reality display region.

Generally, a temporal or spatial fade in the amount of opacity can beused in the opacity filter. Similarly, a temporal or spatial fade in theaugmented reality image can be used. In one approach, a temporal fade inthe amount of opacity of the opacity filter corresponds to a temporalfade in the augmented reality image. In another approach, a spatial fadein the amount of opacity of the opacity filter corresponds to a spatialfade in the augmented reality image. The boundary can be a boundary ofthe augmented reality display region. The boundary can be peripheral,e.g., extending in the horizontal direction, or vertical. Fading isdiscussed further, e.g., in connection with FIG. 9C.

FIG. 3B depicts details of step 306 of FIG. 3A. In step 320, the opacityfilter control circuit identifies pixels of the opacity filter which arebehind the augmented reality image, e.g., based on the size, shape andlocation of the augmented reality image. A variety of approaches arepossible. In one approach, at step 322, an increased opacity is providedfor the pixels of the opacity filter which are behind the augmentedreality image, from the perspective of the identified location of theuser's eye. In this manner, the pixels behind the augmented realityimage are darkened so that light from a corresponding portion of thereal world scene is blocked from reaching the user's eyes. This allowsthe augmented reality image to be realistic and represent a full rangeof colors and intensities. Moreover, power consumption by the augmentedreality emitter is reduced since the augmented reality image can beprovided at a lower intensity. Without the opacity filter, the augmentedreality image would need to be provided at a sufficiently high intensitywhich is brighter than the corresponding portion of the real worldscene, for the augmented reality image to be distinct and nottransparent. In darkening the pixels of the opacity filter, generally,the pixels which follow the closed perimeter of augmented reality imageare darkened, along with pixels within the perimeter. See, e.g., FIGS.4D and 5. It can be desirable to provide some overlap so that somepixels which are outside the perimeter and surround the perimeter arealso darkened. See region 404 in FIG. 4D. These overlapping pixels canprovide a darkened region have a uniform thickness around the perimeter.In another approach, interesting effects can be achieved, e.g., bydarkening all or most of the pixels of the opacity filter which areoutside the perimeter of the augmented reality image, while allowing thepixels within the perimeter of the augmented reality image to remainlight-transmissive.

Step 324 provides an increased opacity for pixels of the opacity filterwhich are outside an augmented reality display region of a field ofview. Generally, the field of view of a user is the angular extent ofthe observable world, vertically and horizontally, that is seen at anygiven moment. Humans have an almost 180-degree forward-facing field ofview. However, the ability to perceive color is greater in the center ofthe field of view, while the ability to perceive shapes and motion isgreater in the periphery of the field of view. Furthermore, asmentioned, the augmented reality image is constrained to being providedin a subset region of the user's field of view. In an exampleimplementation, the augmented reality image is provided in the center ofthe field of view over an angular extent of about 20 degrees, whichlines up with the fovea of the eye. This is the augmented realitydisplay region of the field of view. See, e.g., FIGS. 9A1 and 9B1(region defined by α1) for further details. The augmented reality imageis constrained by factors such as the size of the optical componentsused to route the augmented reality image to the user's eye.

On the other hand, due to its incorporation into the lens, the opacityfilter can extend in a larger range of the field of view, such as about60 degrees, as well as including the first field of view. See, e.g.,FIGS. 9A1 and 9B1 (region defined by α2) for further details. Pixels ofthe opacity filter which are outside the first field of view in theperipheral direction, for instance, can be provided with an increasedopacity in correspondence with an increased opacity for pixels of theopacity filter which are inside the first field of view. See, e.g.,FIGS. 9D1-D3 for further details. This can be useful, e.g., in providinga peripheral cue which accentuates movement of the augmented realityimage, for instance. For example, the peripheral cue may appear as ashadow of the augmented reality image. The peripheral cue may or may notbe in a region of peripheral vision of the user. The peripheral cue canenhance a sense of movement or otherwise capture the user's attention.

Further, when the augmented reality image is near a boundary of theaugmented reality display region of the field of view, correspondingpixels of the opacity filter which are outside the field of view can beprovided with an increased opacity uniformly, or in a spatial fade. Forexample, the increased-opacity pixels can be adjacent to the augmentedreality image at the boundary. The augmented reality image can be afirst portion of an image, where a second portion of the image is cutoffat the boundary, so that it is not displayed, in which case theincreased-opacity pixels can represent the second portion of the image,having a similar size and shape as the second portion of the image. See,e.g., FIGS. 9E1-9E3 for further details. In some cases, theincreased-opacity pixels can have a similar color as the second portionof the image.

Even if the augmented reality image is not cutoff at the boundary, theincreased-opacity pixels can be provided to represent a transition fromthe augmented reality image to the real world scene. In one approach,the increased-opacity pixels are faded so that the pixels of the opacityfilter which are closer to the augmented reality image at the boundaryare more opaque, and the pixels of the opacity filter which are furtherfrom the augmented reality image at the boundary are morelight-transmissive.

Another option involves providing an increased-opacity for pixels of theopacity filter at a time when an augmented reality image is not present,such as to provide a peripheral or non-peripheral cue. Such a cue mightbe useful in an application in which there is motion in the real-worldscene, for instance. Or, the increased-opacity pixels of the opacityfilter can provide a representation of the augmented reality image in aperipheral region of the field of view. See, e.g., FIGS. 9F1-9F3 forfurther details.

Step 326 provides a gradual transition in opacity, e.g., a spatial fade,when the augmented reality image is near a boundary of the augmentedreality display region of the field of view. To avoid an abrupttransition in the augmented reality image, a spatial fade in theaugmented reality image can occur such as described in step 314. Acorresponding fade can occur in the pixels of the opacity filter. Forexample, the augmented reality image can become more faded, and thepixels of the opacity filter can become less opaque, for portions of theaugmented reality image which are closer to the boundary than forportions of the augmented reality image which are further from theboundary. A gradual transition in opacity can similarly be provided evenif the augmented reality image is not near a boundary of the augmentedreality display region of the field of view.

FIG. 4A depicts an example configuration of an opacity filter 400 basedon a shape of the augmented reality image of FIG. 4C. The opacity filterprovides a region 402 of increased opacity. An increased opacitygenerally refers to a darkening of pixels which can include a darkeningto different grey levels in a monochrome scheme, or a darkening todifferent color levels in a color scheme.

FIG. 4B depicts the example real-world scene 120 of FIG. 1. When lightfrom the real-world scene 120 passes through the opacity filter, thelight is multiplied by the opacity filter 400 such thatincreased-opacity area multiplies the corresponding area of thereal-world scene by a “0,” so that the corresponding area of thereal-world scene is not transmitted through the opacity filter, whilethe non-darkened area multiplies the corresponding area of thereal-world scene by a “1,” so that the corresponding area of thereal-world scene is transmitted through the opacity filter.

FIG. 4C depicts the example augmented reality image 104 of FIG. 1. Theaugmented reality image 104 can be rendered with colors and textureswhich are not depicted in this example.

FIG. 4D depicts the example image 132 of FIG. 1 which is seen by a user.The image 132 is formed by adding the image 104 to an image which isformed by multiplying the images 402 and 120. A darkened region 404surrounds the augmented reality image of a dolphin.

FIG. 5 depicts an opacity filter 500 with increased-opacity regions, toprovide the configuration of the opacity filter of FIG. 4A. Each smallcircle represents a pixel of the opacity filter. Selected pixels whichcorrespond to the size, shape and location of the augmented realityimage are controlled to have an increased opacity. An outline of theaugmented reality image is superimposed for reference.

FIG. 6 depicts a variation of the example image of FIG. 1 which wouldresult without the opacity filter. In this image 600, the augmentedreality image 104 appears to be transparent or ghosted, so that thereal-world scene is visible behind the augmented reality image. Thisresult is less realistic.

FIG. 7A depicts an example implementation of the display device of FIG.1, as worn on a user's head 700. In this example, the frame is similarto a conventional eyeglasses frame and can be worn with a similarcomfort level. However, other implementations are possible, such as aface shield which is mounted to the user's head by a helmet, strap orother means. The frame includes a frame front 702 and temples 704 and705. The frame front holds a see-through lens 701 for the user's lefteye and a see-through lens 703 for the user's right eye. The left andright orientations are from the user's perspective. The left-sidesee-through lens 701 includes a light-transmissive opacity filter 723and a light-transmissive optical component 722 such as a beam splitterwhich mixes an augmented reality image with light from the real-worldscene for viewing by the left eye 706. An opening 724 in the opacityfilter can be provided to allow an eye tracking component 726 to imagethe left eye 706, including the pupil 707. The opening can be, e.g., ahole in the lens 701, or a region of the lens 701 in which the opacityfilter is not provided. The opacity filter can be provided in or onanother light-transmissive lens material such as glass or plastic, asmentioned. Infrared light used by the eye tracking component 726 canpass through such a light-transmissive lens material.

The eye tracking component 726 includes an IR emitter 728 which emits IRlight 730 and an IR sensor 734 which senses reflected IR light 732. Theeye tracking component 726 can be mounted to the frame via an arm 736,in one possible approach.

The right-side see-through lens 701 includes a light-transmissiveopacity filter 721 and an optical component 720 such as a beam splitterwhich mixes an augmented reality image with light from the real-worldscene for viewing by the right eye 718. A right-side augmented realityemitter 716 is mounted to the frame via an arm 714, and a left-sideaugmented reality emitter 708 is mounted to the frame via an arm 710. Anopacity filter control circuit 712 can be mounted to the bridge of theframe, and shared by the left- and right-side opacity filters.Appropriate electrical connections can be made via conductive paths inthe frame, for instance.

FIG. 7B depicts further details of the HMD device of FIG. 7A. Thedisplay device is shown from a perspective of the user looking forward,so that the right-side lens 703 and the left-side lens 701 are depicted.The right-side augmented reality emitter 716 includes a light-emittingportion 762 such as a grid of pixels, and a portion 760 which mayinclude circuitry for controlling the light-emitting portion 762.Similarly, the left-side augmented reality emitter 708 includes alight-emitting portion 742 and a portion 740 with circuitry forcontrolling the light-emitting portion 742. Each of the opticalcomponents 720 and 722 may have the same dimensions, in one approach,including a width w1 and a height h1. The right-side optical component720 includes a top surface 764 through which light enters from theright-side augmented reality emitter 716, an angled half-mirroredsurface 766 within the optical component 720, and a face 768. Light fromthe right-side augmented reality emitter 716 and from portions of thereal-world scene (represented by ray 780) which are not blocked by theopacity filter 770 pass through the face 768 and enter the user'sright-side eye. Similarly, the left-side optical component 722 includesa top surface 744 through which light enters from the left-sideaugmented reality emitter 708, an angled half-mirrored surface 746within the optical component 722, and a face 748. Light from theleft-side augmented reality emitter 708 and from portions of thereal-world scene (represented by ray 771) which are not blocked by theopacity filter 750 pass through the face 748 and enter the user'sleft-side eye. Each of the opacity filters 750 and 770 may have the samedimensions, in one approach, including a width w2>w1 and a height h2>h1.

Typically, the same augmented reality image is provided to both eyes,although it is possible to provide a separate image to each eye such asfor a stereoscopic effect. In an alternative implementation, only oneaugmented reality emitter is routed by appropriate optical components toboth eyes.

FIG. 7C depicts an alternative implementation of the display device ofFIG. 1, as worn on a user's head, where the eye tracking component 790is directly on, and inside, the front eye glass frame 702. In thisimplementation, the eye tracking component does not need to projectthrough the lens 701. The eye tracking component 790 includes an IRemitter 791 which emits IR light 792 and an IR sensor 794 which sensesreflected IR light 793.

Regarding eye tracking, in most cases, it is sufficient to know thedisplacement of the augmented reality glasses relative to the eyes asthe glasses bounce around during motion. The rotation of the eyes (e.g.,the movement of the pupil within the eye socket) is often lessconsequential. Although the alignment of the opacity region and theaugmented reality image is a function of the eye position as well, inpractice, we can align the left side of the opacity display as if theuser was looking left, and the right side of the opacity display as theuser was looking right at the same time by stretching the opacity imageto match both criteria. If we do this, then eye angle can be ignored. Adisadvantage to this approach is that the left side will be wrong whenthe user looks right, and the right side will be wrong when the userlooks left, but the user will not notice, since the user can onlyvisually measure the part that falls into the center of the user's fieldof view.

FIG. 8A1 depicts a registration of a real-world image and anincreased-opacity region of an opacity filter when the user's eye is ina first location relative to a frame of the HMD device. A top view isdepicted. As mentioned, an eye tracking component can be used toidentify a location of the eye relative to the frame. In this example, ahorizontal position of the frame relative to the eye 706 and its pupil707 is considered. The opacity filter 750 and optical component 722 aremounted to the frame and therefore move with the frame. Here, the eye706 is looking straight ahead at an element 800 of a real-world scene,as represented by a line of sight 802. The element 800 has a width xrw.The opacity filter 750 includes a region 804 with an increased opacity,while the optical component 722 includes a corresponding region 805 inwhich an augmented reality image is provided. The regions 804 and 805are assumed to have a width of x3. In practice, the width of the opacityfilter region 804 may be slightly wider than that of the augmentedreality image portion 805. Further, the opacity filter region 804 is ata distance of x1 from a left side of the opacity filter, and at adistance x2 from a right side of the opacity filter. Thus, x1+x2+x3=w2.The augmented reality image portion 805 is at a distance of x4 from aleft side of the optical component 722, and at a distance x5 from aright side of the optical component 722. Thus, x4+x5+x3=w1. The element800 of the real-world scene has a width xrw>x3 and is blocked fromreaching the eye 706 by the region 804 of the opacity filter.

FIG. 8A2 depicts a front-facing view of the real-world scene element 800of FIG. 8A1.

FIG. 8A3 depicts a front-facing view of the opacity filter region 804 ofFIG. 8A1. FIG. 8A4 depicts a front-facing view of the augmented realityimage region 805 of FIG. 8A1.

FIG. 8B1 depicts a registration of a real-world image and anincreased-opacity region of an opacity filter when the user's eye is ina second location relative to a frame of the HMD device. A top view isdepicted. In this example, the frame is shifted to the left relative tothe eye 706. The opacity filter 750 includes a region 806 with anincreased opacity, while the optical component 722 includes acorresponding region 807 in which an augmented reality image isprovided. The regions 806 and 807 are assumed to have a width of x3.Further, the opacity filter region 806 is at a distance of x1′>x1 from aleft side of the opacity filter, and at a distance x ′<x2 from a rightside of the opacity filter. Thus, x1′+x2′+x3=w2. The augmented realityimage portion 807 is at a distance of x4′ from a left side of theoptical component 722, and at a distance x5′ from a right side of theoptical component 722. Thus, x4′+x5′+x3=w1. Also, due to the shift,x4′>x4 and x5′<x5 in this example.

The element 800 of the real-world scene has a width xrw>x3 and isblocked from reaching the eye 706 by the region 806 of the opacityfilter. By detecting the movement of the frame, the locations of theopacity region and/or the augmented reality image can be adjustedaccordingly, such as by being shifted horizontally and/or vertically,while the user maintains a fixed line of sight to a real-world scene.This ensures that the augmented reality image appears in the samelocation of the real-world scene. The opacity region and the augmentedreality image continue to be aligned or registered with one another andwith the real-world scene.

In practice, since the increased-opacity region of the opacity filterappears to be closer to the eye than the distantly-focused augmentedreality image, any change in the position of the increased-opacityregion of the opacity filter is more noticeable compared to a similarchange in the position of the augmented reality image. This is due to agreater parallax effect for the increased-opacity region of the opacityfilter. Accordingly, an acceptable result can be obtained in many casesby adjusting a position of the increased-opacity region of the opacityfilter without adjusting a position of the augmented reality image,based on the eye tracking. A shift in the position of theincreased-opacity region of the opacity filter can be the same orsimilar to the shift in the location of the eye relative to the frame. Ashift in the position of the augmented reality image can be a smallfraction of the shift in the position of the increased-opacity region ofthe opacity filter.

Another point is that when the user is looking to the right, theleft-side see-through lens and augmented reality image is not focusedon, so that it may be sufficient to adjust the position of theincreased-opacity region of the opacity filter, based on the eyetracking, for the right-side opacity filter only, and not the left-sideopacity filter. Similarly, when the user is looking to the left, theright-side see-through lens and augmented reality image is not focusedon, so that it may be sufficient to adjust the position of theincreased-opacity region of the opacity filter, based on the eyetracking, for the left-side opacity filter only, and not the right-sideopacity filter.

FIG. 8B2 depicts another view of the real-world scene element 800 ofFIG. 8B1.

FIG. 8B3 depicts another view of the opacity filter region 804 of FIG.8B1. FIG. 8B4 depicts another view of the augmented reality image region805 of FIG. 8B1.

FIG. 9A1 depicts a registration of an augmented reality image and anincreased-opacity region of an opacity filter, at a center of anaugmented reality display region of a field of view of a user's eye. Atop view is depicted. The scale of FIGS. 9A1, 9B1, 9D1, 9E1 and 9F1 ismodified from that of FIGS. 8A1 and 8B1 by placing the opacity filter750 and the optical component 722 further from the eye, to show furtherdetail. As discussed, the eye has a field of view which is relativelywide. The opacity filter 750 is within a field of view with an angularextent of α2, such as about 60 degrees, bounded by lines 904 and 912,and the optical component 722, which provides the augmented realityimage, is within a field of view with an angular extent of α1, such asabout 20 degrees, bounded by lines 906 and 910. The field of view withan angular extent of α1 represents an angular extent of the augmentedreality display region. Line 908 represents a straight ahead line ofsight of the eye, which passes through a center of the augmented realityimage 900 and the increased-opacity region 902. Further, a portion 902of the opacity filter has an increased opacity and a correspondingportion of the optical component 900 provides the augmented realityimage. The increased-opacity portion 902 of the opacity filter is behindthe augmented reality image. This example depicts the augmented realityimage and the increased-opacity region of the opacity filter being atthe center of the augmented reality display region, and not at aboundary of the augmented reality display region (represented byboundary lines 906 and 910).

In one approach, the opacity filter has the ability to provide color, sothat a high resolution, color image is provided in the central 20 degree(=/−10 degrees to the left and right) field of view, while theperipheral region (between +/−10 to 30 degrees to the left and right)uses the opacity filter to provide an increased opacity and color but ata lower resolution, and out of focus. For example, as the user moves hishead side to side, we can adjust the position of the augmented realityimage, such as the flying dolphin, so that the dolphin can move from thecentral 20 degree field of view to the peripheral regions, where theopacity filter represents the augmented reality image. This avoids adiscontinuity which would result if the dolphin disappeared when itmoved out of the central 20 degree field of view.

In this and the following figures, the augmented reality image and theincreased-opacity regions are assumed to have a corresponding square orrectangular shape, for simplicity.

FIG. 9A2 depicts a front-facing view of the opacity filter region 902 ofFIG. 9A1. FIG. 9A3 depicts a front-facing view of the augmented realityimage region 900 of FIG. 9A1. FIG. 9B1 depicts a registration of anaugmented reality image and an increased-opacity region of an opacityfilter, at a peripheral boundary of the augmented reality display regionof FIG. 9A1. Here, the augmented reality image 922 and theincreased-opacity region 920 (both assumed to have a width of about d2)of the opacity filter are at the boundary 906 of the augmented realitydisplay region (represented by boundary lines 906 and 910). A line 907represents a line of sight through a center of the augmented realityimage 922 and the increased-opacity region 920. α3 is an angular extentbetween lines 906 and 907.

As mentioned in connection with FIGS. 3A and 3B, it is possible toprovide a gradual transition in opacity such as when the augmentedreality image is at a boundary of the augmented reality display region,as is the case in FIG. 9B1. See below for further details of such atransition.

FIG. 9B2 depicts a front-facing view of the opacity filter region 920 ofFIG. 9B1. FIG. 9B3 depicts a front-facing view of the augmented realityimage region 922 of FIG. 9B1.

FIG. 9C1 depicts a gradual change in opacity as a function of a distancefrom a peripheral boundary of a field of view. The x-axis represents ahorizontal distance from the boundary line 906 and the y-axis representsan opacity of a corresponding region of the opacity filter. In oneoption, represented by line 915, the opacity is at a maximum level at adistance of d1 to d2 from the boundary and decreases gradually to aminimum level at the boundary (x=0). See FIG. 9C2. The opacity is at theminimum level for x<0, outside the augmented reality display region. Inanother option, represented by line 916, the opacity is at a maximumlevel from x=0 to x=d2 from the boundary and decreases gradually outsidethe augmented reality display region to a minimum level over a distance|d3| from the boundary. See FIG. 9C3. The opacity is at the minimumlevel for x<d3, outside the augmented reality display region. In yetanother option, represented by line 917, the opacity is at a maximumlevel from x=d4 to x=d2 and decreases gradually outside the augmentedreality display region to a minimum level over a distance |d5|-|d4| SeeFIG. 9C4. The opacity is at the minimum level for x<d5, outside theaugmented reality display region.

FIG. 9C2 depicts an opacity filter region with a non-faded portion 931and successively faded portions 932, 933 and 934, with fading between 0and d1 in FIG. 9C1.

FIG. 9C3 depicts an opacity filter region with a non-faded portion 941and successively faded portions 942, 943 and 944, with fading between 0and d3 in FIG. 9C1.

FIG. 9C4 depicts an opacity filter region with a non-faded portion 951and successively faded portions 952, 953 and 954, with fading between d4and d5 in FIG. 9C1.

FIG. 9D1 depicts a registration of an augmented reality image and anincreased-opacity region of an opacity filter, at a peripheral boundaryof the augmented reality display region of FIG. 9A1, where an additionalregion of increased opacity is provided in a second, peripheral regionof the field of view. A top view is depicted. Compared to FIG. 9B1, FIG.9D1 adds an additional increased-opacity region 924 of the opacityfilter 750. The additional increased-opacity region 924, which isoutside the augmented reality display region, can provide a peripheralcue such as a shadow for the augmented reality image 922. The shadow canhave a similar size and shape as the augmented reality image 922. Theadditional increased-opacity region 924 can be on the same levelhorizontally and/or above or below the augmented reality image 922and/or the increased-opacity region 920. In this example, theincreased-opacity region 924 is separated from the increased-opacityregion 920 by a transmissive region of the opacity filter 750.

The second, peripheral region of the field of view, on a left peripheralside of the optical component 722, has an angular extent of (α2−α1)/2(e.g., 10-30 degrees) between lines 904 and 906 on a left peripheralside of the optical component 722. A corresponding additional peripheralregion has an angular extent of (α2−α1)/2 between lines 910 and 912 on aright peripheral side of the optical component 722.

FIG. 9D2 depicts a front-facing view of the opacity filter regions 920and 924 of FIG. 9D1.

FIG. 9D3 depicts a front-facing view of the augmented reality imageregion 900 of FIG. 9D1.

FIG. 9E1 depicts a registration of a first portion of an augmentedreality image and an increased-opacity region of an opacity filter, at aperipheral boundary of the augmented reality display region of FIG. 9A1,where an additional region of increased opacity is provided in a second,peripheral region of the field of view to represent a second, cutoffportion of the augmented reality image. A top view is depicted. Here,the augmented reality image portion 922, with width d2, represents afirst portion of the augmented reality image, and an increased-opacityregion 926 of the opacity filter 750 is behind the augmented realityimage portion 922. An augmented reality image portion 923, of width d2′,which is not actually present, represents where a second, cutoff portionof the augmented reality image would be located, based on the positionof the augmented reality image portion 922. In this case, an additionalincreased-opacity region 928 of the opacity filter 750 (which can be acontinuation of the increased-opacity region 926) is provided behind theaugmented reality image portion 923 to avoid an abrupt cutoff in theaugmented reality image. The additional increased-opacity region 928 canend with a step change to a minimum opacity, or can be provided with agradual change in opacity, using an approach which is analogous to thediscussion of FIG. 9C1.

In one approach, the additional increased-opacity region 928 has asimilar size, shape location and/or color as the augmented reality imageportion 923, so that it essentially represents the augmented realityimage portion 923 which is not actually present.

FIG. 9E2 depicts a front-facing view of the opacity filter regions 926and 928 of FIG. 9E1.

FIG. 9E3 depicts a front-facing view of the augmented reality imageregions 922 and 923 of FIG. 9E1.

FIG. 9F1 depicts an increased-opacity region 960 of an opacity filter750 in a second, peripheral region of a field of view, at a time when noaugmented reality image is provided by the optical component 722. A topview is depicted. One or more increased-opacity regions can be providedin either peripheral region, on the right or left side.

As discussed, the increased-opacity region 960 can represent a lowerresolution and out-of-focus version of the augmented reality image. Thiscan be useful, e.g., when the user moves his head to the side so thatthe augmented reality image moves out of the central 20 degree field ofview to a peripheral region of the field of view. This movement could berepresented by the sequence of FIG. 9A1, where the augmented realityimage 900 is in the central field of view, followed by the FIG. 9B1,where the augmented reality image 922 is at a boundary of the centralfield of view, followed by the FIG. 9F1, where the opaque region 960(representing the augmented reality image) is in the peripheral regionof the field of view. As the user moves his head back to the startingposition, the sequence can be reversed.

The increased-opacity pixels of the opacity filter in the peripheralregion can have a corresponding shape as the augmented reality image,and/or a corresponding color when the opacity filter has a colorcapability. The positioning and timing of the increased-opacity pixelsof the opacity filter can be set to provide a smooth transition based onmovement of the augmented reality image. For example, as the augmentedreality image reaches the boundary of the central field of view, theopacity filter can be activated accordingly to provide a correspondingshape and movement in the peripheral region as a representation of theaugmented reality image. Subsequently, as the representation of theaugmented reality image moves toward the boundary of the central fieldof view, the opacity filter can be deactivated and the augmented realityimage can be activated accordingly to provide a corresponding shape andmovement in the central field of view.

FIG. 9F2 depicts a front-facing view of the opacity filter region 960 ofFIG. 9F1.

FIG. 9F3 depicts a front-facing view of the augmented reality image ofFIG. 9F1.

As can be seen, a number of advantages are provided. For example, arelatively streamlined HMD apparatus is provided. Furthermore,calibration between the eye, the primary color display, and the opacityfilter is provided using eye tracking and psycho-perceptual techniques.The opacity filter can be used to provide peripheral vision cues evenwhere there is no primary display providing virtual imagery. Forcolor-based opacity filters, we can seamlessly blend the peripheralcolor area with the central focus area for a better overall experience,and transition to opacity-only-filtering inside the focus area.

The foregoing detailed description of the technology herein has beenpresented for purposes of illustration and description. It is notintended to be exhaustive or to limit the technology to the precise formdisclosed. Many modifications and variations are possible in light ofthe above teaching. The described embodiments were chosen to bestexplain the principles of the technology and its practical applicationto thereby enable others skilled in the art to best utilize thetechnology in various embodiments and with various modifications as aresuited to the particular use contemplated. It is intended that the scopeof the technology be defined by the claims appended hereto.

What is claimed is:
 1. An optical see-through head-mounted displaydevice comprising: a see-though lens extending between a user's eye anda real-world scene when the display is worn by the user, the see-thoughlens comprising an opacity filter which can be controlled to selectivelytransmit or block light on a per-pixel basis, and a display component; amicro-display configured to emit light to the user's eye using thedisplay component, wherein the light represents an augmented-realityimage having a shape; and, an opacity filter control circuit configuredto control the opacity filter to provide an increased opacity for pixelswhich follow a perimeter of the shape of the augmented-reality image,for pixels which are within the perimeter of the shape, and for pixelswhich surround the perimeter in a region of uniform thickness around theperimeter, from a perspective of the user's eye.
 2. The opticalsee-through head-mounted display device of claim 1, wherein: thesee-through lens is mounted to a frame worn on the user's head; theoptical see-through head-mounted display device further comprises atracking component which tracks a location of the user's eye relative tothe frame; and, the opacity filter control circuit is responsive to thelocation of the user's eye relative to the frame.
 3. The opticalsee-through head-mounted display device of claim 2, wherein the trackingcomponent is mounted to the frame and tracks a center of a pupil of theeye.
 4. The optical see-through head-mounted display device of claim 1,wherein the see-through lens is mounted to a frame front of a frame wornon the user's head.
 5. The optical see-through head-mounted displaydevice of claim 2, wherein the opacity filter control circuit isconfigured to maintain a registration of the pixels which are behind theaugmented reality image with the augmented reality image incorrespondence with the location of the user's eye relative to theframe.
 6. The optical see-through head-mounted display device of claim2, wherein the at least one micro-display maintains a registration ofthe augmented reality image with the pixels which are behind theaugmented reality image in correspondence with the location of theuser's eye relative to the frame.
 7. The optical see-throughhead-mounted display device of claim 1, wherein the display componentcomprises at least one optical component which combines the light fromthe real-world scene and the light representing the augmented realityimage, the display component is between the opacity filter and theuser's eye.
 8. The optical see-through head-mounted display device ofclaim 1, wherein the opacity filter is out of focus to the user's eyedue to the opacity filter being at a near distance to the user eye, sothat a fuzzy black border surrounds the shape of the augmented realityimage; and, the augmented reality image is in focus to the user's eye.9. The optical see-through head-mounted display device of claim 1,wherein the pixels of the opacity filter which surround the perimeter inthe region of uniform thickness around the perimeter provide a darkenedregion around the augmented reality image, the darkened region has ashape corresponding to the shape of the augmented reality image.
 10. Theoptical see-through head-mounted display device of claim 1, wherein theaugmented reality image is limited to a first angular extent of a fieldof view of the user' eye; and, the opacity filter extends in a secondangular extent which includes the first angular extent and a moreperipheral angular extent of the field of view.
 11. The opticalsee-through head-mounted display device of claim 10, wherein the opacityfilter control circuit is configured to provide an increased opacity forpixels of the opacity filter which are in the more peripheral angularextent to depict a peripheral cue for the augmented reality image. 12.The optical see-through head-mounted display device of claim 10, whereinthe opacity filter control circuit is configured to provide an increasedopacity for pixels of the opacity filter which are in the moreperipheral angular extent to depict a representation of the augmentedreality image.
 13. The optical see-through head-mounted display deviceof claim 1, wherein the opacity filter comprises an LCD panel.
 14. Theoptical see-through head-mounted display device of claim 1, wherein theopacity filter comprises an electrochromic film.
 15. The opticalsee-through head-mounted display device of claim 1, wherein the opacityfilter control circuit is configured to control the opacity filter toprovide non-darkened pixels of the opacity filter which are not behindthe augmented reality image.
 16. The optical see-through head-mounteddisplay device of claim 7, wherein the at least one optical componentcomprises a beam splitter adjacent to the opacity filter.
 17. Theoptical see-through head-mounted display device of claim 1, wherein theincreased opacities are more opaque than an opacity of other pixels ofthe opacity filter.